Run CI tests in parallel #25271
Run CI tests in parallel #25271
Conversation
dantecatalfamo
had a problem deploying
to
Docker Hub
GitHub Actions
Error
dantecatalfamo
had a problem deploying
to
Docker Hub
GitHub Actions
Error
dantecatalfamo
temporarily deployed
to
Docker Hub
GitHub Actions
Inactive
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #25271 +/- ##
==========================================
- Coverage 63.85% 63.83% -0.03%
==========================================
Files 1616 1616
Lines 153836 153851 +15
Branches 3976 3976
==========================================
- Hits 98230 98207 -23
- Misses 47791 47838 +47
+ Partials 7815 7806 -9
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
dantecatalfamo
temporarily deployed
to
Docker Hub
GitHub Actions
Inactive
dantecatalfamo
requested review from
lukeheath,
georgekarrv and
sharon-fdm
as code owners
dantecatalfamo
temporarily deployed
to
Docker Hub
GitHub Actions
Inactive
|
@lucasmrod I'm running a new test right now with all of |
|
It's about the same, the longest test is still the integration test. In the future we could run the Enterprise and MDM integration tests as their own unit to speed it up |
| PKG_TO_TEST := "" # default to empty string; can be overridden on command line. | ||
| go_test_pkg_to_test := $(addprefix ./,$(PKG_TO_TEST)) # set paths for packages to test | ||
| dlv_test_pkg_to_test := $(addprefix github.com/fleetdm/fleet/v4/,$(PKG_TO_TEST)) # set URIs for packages to debug | ||
|
|
||
| DEFAULT_PKG_TO_TEST := ./cmd/... ./ee/... ./orbit/pkg/... ./orbit/cmd/orbit ./pkg/... ./server/... ./tools/... | ||
| ifeq ($(CI_TEST_PKG), main) | ||
| CI_PKG_TO_TEST=$(shell go list ${DEFAULT_PKG_TO_TEST} | grep -v "server/datastore/mysql" | grep -v "cmd/fleetctl" | grep -v "server/vulnerabilities" | sed -e 's|github.com/fleetdm/fleet/v4/||g') |
There was a problem hiding this comment.
I don't think there's a limit to the # of packages you can specify for go test, so as long as it fits in one shell command we should be 👍
dantecatalfamo
temporarily deployed
to
Docker Hub
GitHub Actions
Inactive
|
I think if we split up the different integration tests into their own units we could save time as well, but I also don't know how far down that rabbit hole I want to go due to potential github action costs. If we could somehow do the schema dump and mock generation in their own action and then reuse it in the other actions, we could probably save ~1.5m per test. Maybe something to look into in another ticket. |
Follow up to #25271 and #21774 Integration test failures will happen much faster of they occur, but now the bottleneck is the `fleetctl` test suite. It's trivial to continue splitting tests up now. We should look into creating an action that checks that mock generation is up-to-date, run it before all the tests, and then remove the mock generation step from each test step. That would save about a minute and a half of runtime from each test and help offset the cost of splitting the tests up.   
#21774
Improves run time by about 30%.
Things have been arranged in such a way that splitting modules out further will be trivial in the future, such as breaking the different integration test suited into their own units.